<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-us" lang="en-us">
<head>
  <link href="//gmpg.org/xfn/11" rel="profile">
  <meta http-equiv="content-type" content="text/html; charset=utf-8">
  <meta name="generator" content="Hugo 0.68.3" />

  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">

  <title>全文检索 &middot; 我的博客</title>

  
  <link type="text/css" rel="stylesheet" href="/my_technology_blog/css/print.css" media="print">
  <link type="text/css" rel="stylesheet" href="/my_technology_blog/css/poole.css">
  <link type="text/css" rel="stylesheet" href="/my_technology_blog/css/syntax.css">
  <link type="text/css" rel="stylesheet" href="/my_technology_blog/css/hyde.css">
    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Abril+Fatface|PT+Sans:400,400i,700">


  
  <link rel="apple-touch-icon-precomposed" sizes="144x144" href="/apple-touch-icon-144-precomposed.png">
  <link rel="shortcut icon" href="/favicon.png">

  
  
</head>

  <body class="theme-base-0b ">
  <aside class="sidebar">
  <div class="container sidebar-sticky">
    <div class="sidebar-about">
      <a href="/my_technology_blog/"><h1>我的博客</h1></a>
      <p class="lead">
       杨博的博客 
      </p>
    </div>

    <nav>
      <ul class="sidebar-nav">
        <li><a href="/my_technology_blog/">Home</a> </li>
        
      </ul>
    </nav>

    <p>&copy; 2021. All rights reserved. </p>
  </div>
</aside>

    <main class="content container">
    <div class="post">
  <h1>全文检索</h1>
  <time datetime=2020-05-12T10:30:32&#43;0800 class="post-date">Tue, May 12, 2020</time>
  <p>全文检索django-haystack+jieba+whoosh</p>
<p><strong>全文检索里的组件简介</strong></p>
<p>　　1、什么是haystack？</p>
<p>　　　　1. haystack是django的开源搜索框架，该框架支持Solr,Elasticsearch,Whoosh, Xapian搜索引擎，不用更改代码，直接切换引擎，减少代码量。</p>
<p>　　　　2. 搜索引擎使用Whoosh，这是一个由纯Python实现的全文搜索引擎，没有二进制文件等，比较小巧，配置比较简单，当然性能自然略低。</p>
<p>　　　　3. 中文分词Jieba，由于Whoosh自带的是英文分词，对中文的分词支持不是太好，故用jieba替换whoosh的分词组件。</p>
<p>　　2、什么是jieba？</p>
<p>　　　　1、很多的搜索引擎对中的支持不友好，jieba作为一个中文分词器就是加强对中文的检索功能</p>
<p>　　3、Whoosh是什么？</p>
<p>　　　　1、Python的全文搜索库，Whoosh是索引文本及搜索文本的类和函数库</p>
<p>　　　　2、Whoosh 自带的是英文分词，对中文分词支持不太好，使用 jieba 替换 whoosh 的分词组件。</p>
<p>haystack配置使用（前后端分离）</p>
<p>安装工具</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">pip install django<span style="color:#f92672">-</span>haystack
pip install whoosh
pip install jieba
</code></pre></div><p>　　在setting.py中配置</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#e6db74">&#39;&#39;&#39;注册app &#39;&#39;&#39;</span>
INSTALLED_APPS <span style="color:#f92672">=</span> [
    <span style="color:#e6db74">&#39;django.contrib.admin&#39;</span>,
    <span style="color:#e6db74">&#39;django.contrib.auth&#39;</span>,
    <span style="color:#e6db74">&#39;django.contrib.contenttypes&#39;</span>,
    <span style="color:#e6db74">&#39;django.contrib.sessions&#39;</span>,
    <span style="color:#e6db74">&#39;django.contrib.messages&#39;</span>,
    <span style="color:#e6db74">&#39;django.contrib.staticfiles&#39;</span>,
    <span style="color:#75715e"># haystack要放在应用的上面</span>
    <span style="color:#e6db74">&#39;haystack&#39;</span>,
    <span style="color:#e6db74">&#39;jsapp&#39;</span>,  <span style="color:#75715e"># 这个jsapp是自己创建的app</span>
]
</code></pre></div><p>​</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#e6db74">&#39;&#39;&#39; 模板路径 &#39;&#39;&#39;</span>
TEMPLATES <span style="color:#f92672">=</span> [
    {
        <span style="color:#e6db74">&#39;DIRS&#39;</span>: [os<span style="color:#f92672">.</span>path<span style="color:#f92672">.</span>join(BASE_DIR,<span style="color:#e6db74">&#39;templates&#39;</span>)],

    },
]
</code></pre></div><p>​</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#e6db74">&#39;&#39;&#39;配置haystack &#39;&#39;&#39;</span>
<span style="color:#75715e"># 全文检索框架配置</span>
HAYSTACK_CONNECTIONS <span style="color:#f92672">=</span> {
    <span style="color:#e6db74">&#39;default&#39;</span>: {
        <span style="color:#75715e"># 指定whoosh引擎</span>
        <span style="color:#e6db74">&#39;ENGINE&#39;</span>: <span style="color:#e6db74">&#39;haystack.backends.whoosh_backend.WhooshEngine&#39;</span>,
        <span style="color:#75715e"># &#39;ENGINE&#39;: &#39;jsapp.whoosh_cn_backend.WhooshEngine&#39;,      # whoosh_cn_backend是haystack的whoosh_backend.py改名的文件为了使用jieba分词</span>
        <span style="color:#75715e"># 索引文件路径</span>
        <span style="color:#e6db74">&#39;PATH&#39;</span>: os<span style="color:#f92672">.</span>path<span style="color:#f92672">.</span>join(BASE_DIR, <span style="color:#e6db74">&#39;whoosh_index&#39;</span>),
    }
}
<span style="color:#75715e"># 添加此项，当数据库改变时，会自动更新索引，非常方便</span>
HAYSTACK_SIGNAL_PROCESSOR <span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;haystack.signals.RealtimeSignalProcessor&#39;</span>
</code></pre></div><p>　　定义数据库（用于测试）</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models

<span style="color:#75715e"># Create your models here.</span>
<span style="color:#66d9ef">class</span> <span style="color:#a6e22e">UserInfo</span>(models<span style="color:#f92672">.</span>Model):
    name <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>CharField(max_length<span style="color:#f92672">=</span><span style="color:#ae81ff">254</span>)
    age <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>IntegerField()
</code></pre></div><p>​</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">ArticlePost</span>(models<span style="color:#f92672">.</span>Model):
    author <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>ForeignKey(UserInfo,on_delete<span style="color:#f92672">=</span>models<span style="color:#f92672">.</span>CASCADE)
    title <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>CharField(max_length<span style="color:#f92672">=</span><span style="color:#ae81ff">200</span>)
    desc <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>SlugField(max_length<span style="color:#f92672">=</span><span style="color:#ae81ff">500</span>)
    body <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>TextField()
</code></pre></div><p>索引文件生成</p>
<p>　　<em>1）在子应用下创建索引文件</em></p>
<p>　　　　<em>在子应用的目录下，创建一个名为 jsapp/search_indexes.py 的文件</em></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#75715e">#! /usr/bin/env python</span>
<span style="color:#75715e"># -*- coding: utf-8 -*-</span>
<span style="color:#f92672">from</span> haystack <span style="color:#f92672">import</span> indexes
<span style="color:#f92672">from</span> .models <span style="color:#f92672">import</span> ArticlePost

<span style="color:#75715e"># 修改此处，类名为模型类的名称+Index，比如模型类为GoodsInfo,则这里类名为GoodsInfoIndex(其实可以随便写)</span>
<span style="color:#66d9ef">class</span> <span style="color:#a6e22e">ArticlePostIndex</span>(indexes<span style="color:#f92672">.</span>SearchIndex, indexes<span style="color:#f92672">.</span>Indexable):
    <span style="color:#75715e"># text为索引字段</span>
    <span style="color:#75715e"># document = True，这代表haystack和搜索引擎将使用此字段的内容作为索引进行检索</span>
    <span style="color:#75715e"># use_template=True 指定根据表中的那些字段建立索引文件的说明放在一个文件中</span>
    text <span style="color:#f92672">=</span> indexes<span style="color:#f92672">.</span>CharField(document<span style="color:#f92672">=</span>True, use_template<span style="color:#f92672">=</span>True)

    <span style="color:#75715e"># 对那张表进行查询</span>
    <span style="color:#66d9ef">def</span> <span style="color:#a6e22e">get_model</span>(self):  <span style="color:#75715e"># 重载get_model方法，必须要有！</span>
        <span style="color:#75715e"># 返回这个model</span>
        <span style="color:#66d9ef">return</span> ArticlePost

    <span style="color:#75715e"># 建立索引的数据</span>
    <span style="color:#66d9ef">def</span> <span style="color:#a6e22e">index_queryset</span>(self, using<span style="color:#f92672">=</span>None):
        <span style="color:#75715e"># 这个方法返回什么内容，最终就会对那些方法建立索引，这里是对所有字段建立索引</span>
        <span style="color:#66d9ef">return</span> self<span style="color:#f92672">.</span>get_model()<span style="color:#f92672">.</span>objects<span style="color:#f92672">.</span>all()
</code></pre></div><p>　　2）指定索引模板文件　　　</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#75715e"># 创建文件路径命名必须这个规范：templates/search/indexes/应用名称/模型类名称_text.txt</span>
<span style="color:#75715e"># templates/search/indexes/jsapp/articlepost_text.txt</span>
</code></pre></div><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#75715e"># 在创建的txt文件写入</span>
{{ object<span style="color:#f92672">.</span>title }}
{{ object<span style="color:#f92672">.</span>author<span style="color:#f92672">.</span>name }}
{{ object<span style="color:#f92672">.</span>body }}
</code></pre></div><p>　　</p>
<p>　　3）使用命令创建索引</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">python manage<span style="color:#f92672">.</span>py rebuild_index  <span style="color:#75715e"># 建立索引文件</span>
</code></pre></div><p>替换成jieba分词</p>
<p>　　<em>1）将haystack源码复制到项目中并改名</em></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#e6db74">&#39;&#39;&#39;1.复制源码中文件并改名 &#39;&#39;&#39;</span>
<span style="color:#960050;background-color:#1e0010">将</span> C:\python37\Lib\site<span style="color:#f92672">-</span>packages\haystack\backends\whoosh_backend<span style="color:#f92672">.</span>py文件复制到项目中
<span style="color:#960050;background-color:#1e0010">并将</span> whoosh_backend<span style="color:#f92672">.</span>py改名为 whoosh_cn_backend<span style="color:#f92672">.</span>py <span style="color:#960050;background-color:#1e0010">放在</span>APP中如<span style="color:#960050;background-color:#1e0010">：</span>jsapp\whoosh_cn_backend<span style="color:#f92672">.</span>py

<span style="color:#e6db74">&#39;&#39;&#39;2.修改源码中文件&#39;&#39;&#39;</span>
<span style="color:#75715e"># 在全局引入的最后一行加入jieba分词器</span>
<span style="color:#f92672">from</span> jieba.analyse <span style="color:#f92672">import</span> ChineseAnalyzer

<span style="color:#75715e"># 修改为中文分词法</span>
<span style="color:#960050;background-color:#1e0010">查找</span>
analyzer<span style="color:#f92672">=</span>StemmingAnalyzer()
<span style="color:#960050;background-color:#1e0010">改为</span>
analyzer<span style="color:#f92672">=</span>ChineseAnalyzer()
</code></pre></div><p>　　2）Django内settings内修改相应的haystack后台文件名</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#75715e"># 全文检索框架配置</span>
HAYSTACK_CONNECTIONS <span style="color:#f92672">=</span> {
    <span style="color:#e6db74">&#39;default&#39;</span>: {
        <span style="color:#75715e"># 指定whoosh引擎</span>
        <span style="color:#e6db74">&#39;ENGINE&#39;</span>: <span style="color:#e6db74">&#39;haystack.backends.whoosh_backend.WhooshEngine&#39;</span>,
        <span style="color:#75715e"># &#39;ENGINE&#39;: &#39;jsapp.whoosh_cn_backend.WhooshEngine&#39;,      #article.whoosh_cn_backend便是你刚刚添加的文件</span>
        <span style="color:#75715e"># 索引文件路径</span>
        <span style="color:#e6db74">&#39;PATH&#39;</span>: os<span style="color:#f92672">.</span>path<span style="color:#f92672">.</span>join(BASE_DIR, <span style="color:#e6db74">&#39;whoosh_index&#39;</span>),
    }
}
<span style="color:#75715e"># 添加此项，当数据库改变时，会自动更新索引，非常方便</span>
HAYSTACK_SIGNAL_PROCESSOR <span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;haystack.signals.RealtimeSignalProcessor&#39;</span>
</code></pre></div><p>索引文件使用</p>
<p>jsapp/urls.py</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#f92672">from</span> django.conf.urls <span style="color:#f92672">import</span> url
<span style="color:#f92672">from</span> . <span style="color:#f92672">import</span> views <span style="color:#66d9ef">as</span> view

urlpatterns<span style="color:#f92672">=</span>[
    url(<span style="color:#e6db74">r</span><span style="color:#e6db74">&#39;abc/$&#39;</span>, view<span style="color:#f92672">.</span>basic_search),

]
</code></pre></div><p>jsapp/views.py</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#f92672">from</span> django.shortcuts <span style="color:#f92672">import</span> render

<span style="color:#75715e"># Create your views here.</span>
<span style="color:#f92672">import</span> json
<span style="color:#f92672">from</span> django.conf <span style="color:#f92672">import</span> settings
<span style="color:#f92672">from</span> django.core.paginator <span style="color:#f92672">import</span> InvalidPage, Paginator
<span style="color:#f92672">from</span> django.http <span style="color:#f92672">import</span> Http404, HttpResponse,JsonResponse
<span style="color:#f92672">from</span> haystack.forms <span style="color:#f92672">import</span>  ModelSearchForm
<span style="color:#f92672">from</span> haystack.query <span style="color:#f92672">import</span> EmptySearchQuerySet
RESULTS_PER_PAGE <span style="color:#f92672">=</span> getattr(settings, <span style="color:#e6db74">&#39;HAYSTACK_SEARCH_RESULTS_PER_PAGE&#39;</span>, <span style="color:#ae81ff">20</span>)
</code></pre></div><p>​ <br>
​</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">basic_search</span>(request, load_all<span style="color:#f92672">=</span>True, form_class<span style="color:#f92672">=</span>ModelSearchForm, searchqueryset<span style="color:#f92672">=</span>None, extra_context<span style="color:#f92672">=</span>None, results_per_page<span style="color:#f92672">=</span>None):
    query <span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;&#39;</span>
    results <span style="color:#f92672">=</span> EmptySearchQuerySet()
    <span style="color:#66d9ef">if</span> request<span style="color:#f92672">.</span>GET<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#39;q&#39;</span>):
        form <span style="color:#f92672">=</span> form_class(request<span style="color:#f92672">.</span>GET, searchqueryset<span style="color:#f92672">=</span>searchqueryset, load_all<span style="color:#f92672">=</span>load_all)

        <span style="color:#66d9ef">if</span> form<span style="color:#f92672">.</span>is_valid():
            query <span style="color:#f92672">=</span> form<span style="color:#f92672">.</span>cleaned_data[<span style="color:#e6db74">&#39;q&#39;</span>]
            results <span style="color:#f92672">=</span> form<span style="color:#f92672">.</span>search()
    <span style="color:#66d9ef">else</span>:
        form <span style="color:#f92672">=</span> form_class(searchqueryset<span style="color:#f92672">=</span>searchqueryset, load_all<span style="color:#f92672">=</span>load_all)

    paginator <span style="color:#f92672">=</span> Paginator(results, results_per_page <span style="color:#f92672">or</span> RESULTS_PER_PAGE)
    <span style="color:#66d9ef">try</span>:
        page <span style="color:#f92672">=</span> paginator<span style="color:#f92672">.</span>page(int(request<span style="color:#f92672">.</span>GET<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#39;page&#39;</span>, <span style="color:#ae81ff">1</span>)))
    <span style="color:#66d9ef">except</span> InvalidPage:
        result <span style="color:#f92672">=</span> {<span style="color:#e6db74">&#34;code&#34;</span>: <span style="color:#ae81ff">404</span>, <span style="color:#e6db74">&#34;msg&#34;</span>: <span style="color:#e6db74">&#39;No file found！&#39;</span>, <span style="color:#e6db74">&#34;data&#34;</span>: []}
        <span style="color:#66d9ef">return</span> HttpResponse(json<span style="color:#f92672">.</span>dumps(result), content_type<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;application/json&#34;</span>)

    context <span style="color:#f92672">=</span> {
        <span style="color:#e6db74">&#39;form&#39;</span>: form,
        <span style="color:#e6db74">&#39;page&#39;</span>: page,
        <span style="color:#e6db74">&#39;paginator&#39;</span>: paginator,
        <span style="color:#e6db74">&#39;query&#39;</span>: query,
        <span style="color:#e6db74">&#39;suggestion&#39;</span>: None,
    }
    <span style="color:#66d9ef">if</span> results<span style="color:#f92672">.</span>query<span style="color:#f92672">.</span>backend<span style="color:#f92672">.</span>include_spelling:
        context[<span style="color:#e6db74">&#39;suggestion&#39;</span>] <span style="color:#f92672">=</span> form<span style="color:#f92672">.</span>get_suggestion()

    <span style="color:#66d9ef">if</span> extra_context:
        context<span style="color:#f92672">.</span>update(extra_context)
</code></pre></div><p>​</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">    jsondata <span style="color:#f92672">=</span> []
    <span style="color:#66d9ef">print</span>(len(page<span style="color:#f92672">.</span>object_list))
    <span style="color:#66d9ef">for</span> result <span style="color:#f92672">in</span> page<span style="color:#f92672">.</span>object_list:
        data <span style="color:#f92672">=</span> {
            <span style="color:#e6db74">&#39;pk&#39;</span>: result<span style="color:#f92672">.</span>object<span style="color:#f92672">.</span>pk,
            <span style="color:#e6db74">&#39;title&#39;</span>: result<span style="color:#f92672">.</span>object<span style="color:#f92672">.</span>title,
            <span style="color:#e6db74">&#39;content&#39;</span>: result<span style="color:#f92672">.</span>object<span style="color:#f92672">.</span>body,

        }
        jsondata<span style="color:#f92672">.</span>append(data)
    result <span style="color:#f92672">=</span> {<span style="color:#e6db74">&#34;code&#34;</span>: <span style="color:#ae81ff">200</span>, <span style="color:#e6db74">&#34;msg&#34;</span>: <span style="color:#e6db74">&#39;Search successfully！&#39;</span>, <span style="color:#e6db74">&#34;data&#34;</span>: jsondata}
    <span style="color:#66d9ef">return</span> JsonResponse(result, content_type<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;application/json&#34;</span>)
</code></pre></div><p><strong>路由使用127.0.0.1：8000/abc/?q=123</strong></p>

</div>


    </main>

    
  </body>
</html>
